Gaming with earpiece 3d audio

ABSTRACT

A method of providing an enhanced gaming experience includes steps of sensing environmental audio with at least one microphone of an earpiece, combining a game sound with the environmental audio to provide a composite audio signal, and transducing the composite audio signal at the earpiece. The game sound may be positioned within a three-dimensional audio environment. The composite audio signal may be transduced to provide a three-dimensional audio experience.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/082,235 filed on Oct. 28, 2020 which is a continuation of and claimspriority to U.S. patent application Ser. No. 15/795,863 filed on Oct.27, 2017 now patented as 10,821,361 which claims priority to U.S.Provisional Patent Application No. 62/417,064, filed on Nov. 3, 2016 andall entitled “Gaming with Earpiece 3D Audio”, hereby incorporated byreference in their entireties.

FIELD OF THE INVENTION

The present invention relates to gaming. More particularly, but notexclusively, the present invention relates to enhanced gaming usingwearable devices.

BACKGROUND

One recent game that has enjoyed immense initial popularity has beenPOKÉMON GO from Niantic, Inc. In normal operation a mobile device suchas a phone is used which includes a camera. A user may observe videoacquired by the phone by viewing the display of the phone. As a part ofthe gaming experience, an image may be inserted by the game in front ofa video representation of an actual object.

What is needed are new and improved methods and systems for improvingthe gaming experience, especially those that improve the audioexperience of the game.

SUMMARY

Therefore, it is a primary object, feature, or advantage of the presentinvention to improve over the state of the art.

It is a further object, feature, or advantage of the present inventionto improve gaming experiences.

It is a still further object, feature, or advantage of the presentinvention to improve gaming audio experiences.

One or more of these and/or other objects, features, or advantages ofthe present invention will become apparent from the specification andclaims that follow. No single embodiment need provide each and everyobject, feature, or advantage. Different embodiments may have differentobjects, features, or advantages. Therefore, the present invention isnot to be limited to or by any objects, features, or advantages statedherein.

According to one aspect, a method of providing an enhanced gamingexperience is provided. The method includes sensing environmental audiowith at least one microphone of an earpiece, combining a game sound withthe environmental audio to provide a composite audio signal, andtransducing the composite audio signal at the earpiece. The game soundmay be positioned within a three-dimensional audio environment. Thecomposite audio signal may be transduced to provide a three-dimensionalaudio experience. The method may further include sensing user movementwith at least one inertial sensor of the earpiece. The method mayfurther include generating the game sound based on the user movement.The method may further include communicating the user movement to acomputing device executing a game, determining a game sound based on theuser movement from the computing device, and generating the game soundbased on the user movement.

According to another aspect, a method of providing an enhanced gamingexperience is provided. The method includes maintaining an augmentedreality video environment associated with a camera and video displaydevice, maintaining an augmented reality audio environment associatedwith a set of earpieces comprising microphones and speakers, andgenerating a gaming experience using the augmented reality videoenvironment and the augmented reality audio environment. The set ofearpieces may include one or more inertial sensors. The head position ofa user may be sensed using the one or more inertial sensors. Theaugmented reality video environment may include game images and theaugmented reality audio environment may include game sounds. At least aportion of the game images within the augmented reality videoenvironment may have corresponding game sounds present in the augmentedreality audio environment.

According to another aspect, a method of providing an enhanced gamingexperience includes steps of determining a location of a user, sensingenvironmental audio with at least one microphone of at least oneearpiece, selecting a game sound based on the location of the user,generating the game sound based on relative position between thelocation of the user and a perceived source location associated with thegame sound. The method may further include transducing the game soundand the environmental audio at a speaker of the at least one earpiece.The method may further include selecting a game image based on thelocation of the user and displaying the game image to the user based onthe location of the user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial representation of an enhanced gaming experienceusing audio transparency in accordance with the illustrativeembodiments.

FIG. 2 is a block diagram of audio transparency via the convergence ofenvironmental and game audio as shown in the illustrative embodiments.

FIG. 3 is a flowchart depicting one embodiment of wireless earpiececomponents used to create a composite audio effect.

FIG. 4 is an illustrative example of the production of a composite audioeffect via the user's wireless earpieces, a gaming system, and a scenein the user's surrounding environment.

FIG. 5 is a pictorial representation of wireless earpieces with sensors.

FIG. 6 is a pictorial representation of the wireless communication viathe user, the earpieces worn by the user, a smartphone, and anelectronic device in accordance with the illustrative embodiments.

FIG. 7 is a flowchart of the appearance of the imaging andthree-dimensional audio effects to the user as the user moves inaccordance with the illustrative embodiments.

FIG. 8 is a pictorial representation of an enhanced gaming systemdepicting the interaction between the user and the appearance of thegame sound source as a smart phone tracks the user's location.

FIG. 9 is a pictorial representation of the interaction between theuser's head movements, the inertial sensors of the earpieces, the gamingsystem, and the sounds heard by the user.

DETAILED DESCRIPTION

The present invention relates to an enhanced gaming experience. Currentgaming systems, such as POKÉMON GO, utilize a phone, a camera, and animage inserted in front of an object. In operation a mobile device suchas a phone is used which includes a camera. A user may observe videoacquired by the phone by viewing the display of the phone. As a part ofthe gaming experience, an image may be inserted by the game in front ofa video representation of an actual object on the display of the mobiledevice. Thus, a user may, by viewing the display of the mobile device,see actual video imagery acquired with the camera as well as additionalimagery added as a part of the game. Game imagery may be added based onthe location of the user such as through using GPS location data. Thus,when a user is at a specific location they may be able to view specificgame imagery which is mapped to that location.

According to one aspect, one or more ear pieces or headsets are used toenhance audio aspects of the gaming experience. The one or moreearpieces or headsets may use an AUDIO TRANSPARENCY feature whereambient audio is sensed at one or more microphones of the earpiece andthen reproduced at one or more speakers of the earpiece. Thus, eventhough a user may be wearing the earpiece or headset they are still ableto hear ambient or environmental sound. In addition, game sounds may beadded to the ambient or environmental sounds. The game sounds may bemapped to specific locations. Thus, when the user is at a specificlocation, they may hear a particular sound. In addition, soundprocessing may be performed based on the relative location of the userand the location of the source of the sound. Thus, for example, thesound may be heard when the user is at a set distance from the locationof the source of the sound and the intensity of the sound may increaseas the user approaches the location of the source of the sound.Similarly, 3D sound processing may be performed based on the relativelocation of the user (and both the right and left ears of the user) tothe location of the source of the sound thus, for example, a user mayhear the sound in a manner which provides them with spatial context asto where the source of the sound is located relative to their currentposition.

FIG. 1 displays a user 106 experiencing enhanced gaming with the audiotransparency feature. The object image 20 on the phone 104 represents aclue or object in the game system 94. The phone 104 is not limited to aphone or smart phone, but may encompass other electronic devices,including but not limited to other mobile devices, iPADs, tablets, gameconsoles, etc. FIG. 1 displays two electronic devices intended to showthe detail of two systems or two aspects of a system operatingsimultaneously within one device. The game system 94 may encompass allgaming applications or systems played on or incorporating any electronicdevices. As the user 106 moves, the object image 20 on the phone 104 maymove in response to updated position information received via ageospatial navigation system such as a global positioning (“GPS”) systemreceiver or other location tracking system. Of course, other types ofnavigation systems may be used. For example, a GLOSNASS position may beprovided, a position may be interpolated based on nearby cell phonetowers, a position may be determined based on one or more WiFi routerswithin range, a position may be determined based on a combination of oneor more cell phone towers within range and one or more WiFi routerswithin range, the position may be determined based on sensing movementvia one or more inertial sensors, or the position may be otherwisedetermined. The GPS signal 96 received at a GPS receiver of the phone104 may be used to track the movement of the user 106. The game system94 coordinates the user's movement as determined using the GPS signal 96with the object image 20 on the phone. As the user 106 approaches acertain point, shown in FIG. 1 as the user location 100, the phone mayvibrate or make a sound to alert the user 106. This sound is depicted inFIG. 1 as the game sound 110. Upon hearing the game sound 110, the user106 has options, one of which is to continue the game by touching thescreen of the electronic device 104 to activate the object image 20 orother generic images 112 in the game system 94. The user 106 may alsoapproach a certain point and hear only the environmental sounds, astransmitted clearly from the wireless earpiece speakers to the internalmicrophone. Or, the user may reach a certain location and hear the gamesounds overlapping with the environmental sounds. Actions of the usermay vary depending on the particular gaming experience. In oneembodiment of the enhanced gaming experience, a new sound would beinserted in front of the game sound 110. This new sound, depicted inFIG. 1 as environmental sound 108, would be framed by the position ofthe phone 104 and its sensors 102. This environmental sound 108 may comefrom the surrounding environment of the user 106, from the outsideworld, or it may be a sound generated by another electronic device suchas a phone, an earpiece, or another external device. As suggested in theclaims, in one embodiment, the microphone of the earpiece(s) 120 sensesthe environmental sound 108 or the external sound.

FIG. 2 displays an overview of the audio transparency technology. FIG. 2depicts the convergence of the environmental sound 108 with the gamesound 110 to form the composite audio 105. The wireless earpiece(s) 120,described in further detail later herein sense the environmental sound108. The environmental sound 108 and the game sound 110 are combined,and the user 106 experiences the composite audio 105. This compositeaudio may be considered a feature of AUDIO TRANSPARENCY. The overlap ofsounds extends to sounds from the game combined with sounds from othersources, such as other external devices or other users. Other earpieceusers may participate in the game with the user, and the audio messagesthese users send may overlap with the game sounds or the environmentalsounds in an ambient audio format. External speakers may transmitenvironmental sounds to the microphone of the user's earpieces. At thesame time, the user may hear audio messages sent from other earpieceusers. These audio messages may be restricted or require acceptance fromthe user before the message is relayed, depending on the user's earpiecesettings.

FIG. 3 is a flowchart depicting one use of the earpiece microphones tocreate the composite audio 105. In FIG. 3, the external microphones 122of the earpiece(s) 120 may convey the environmental sounds 108 to theprocessor(s) 128, internal microphones 124, speaker 126, or the wirelesstransceivers 130. As the external microphone 122 conveys theenvironmental sound 108, the internal or bone microphone 124 may conveythe game sound 110. As part of the audio transparency technology, theuser 106 may experience game sounds 110 overlapped with environmentalsounds 108. The composite audio may also be created as the wirelesstransceiver relays audio messages from other earpiece users, and themicrophone conveys environmental sounds. Or the internal microphone ofthe wireless earpiece may convey sounds from other sources, such asmusic saved on an electronic device, or music streamed from a server, asthe wireless transceiver conveys audio messages. The composite audioeffect may also be created by layering the game sounds with music fromanother device and environmental sounds.

FIG. 4 depicts one non-limiting example of this audio transparencytechnology. FIG. 4 shows a user 106 looking at a house 84, a tree 80,and several bushes 82 in front of the house. The audio from the world(in this example, sounds from the house and surrounding foliage), wouldbe transmitted normally to the earpiece or another device as theenvironmental sound 108. As the user discovers the clue or item beingsearched for, shown in FIG. 4 as the object image 20, the game alertsthe user by introducing its own sound, the game sound 110, sonicallypositioned where the item would be able to be detected. No gamingexperiences currently overlay game sounds with environmental sounds. Onetype of enhanced experience is created as game sounds 110 are interlacedwith environmental sounds 108 transmitted using audio transparencytechnology which provides for sensing environmental sounds 108 using oneor more earpieces 120 and then reproducing the environmental sounds 108using one or more microphones of the earpiece. Although the exampleabove describes outdoor sounds, environmental sounds may include allvarieties of surrounding environments, including but not limited to:musical sounds, nightlife, churches, temples, and cathedrals, bells,airport sounds, fairground sounds, the sounds of sports games, travelsounds, the sounds of storms, environmental sounds from museums andlibraries, birdsongs, animal calls, etc. Generic images 112 may alsocreate game sounds different from those created by the object image 20,and these generic sounds may also be interlaced with the game sound 110and the environmental sounds 108 to create another form of the compositeaudio sound 105. Generic images in the game may correspond with theenvironmental sounds. The images in the game may change according to theenvironmental sound the user is hearing. For example, if a user is neara cathedral, the GPS locator or the camera may capture this image. Thenthe earpiece may pick up the sound of the bell from the surroundingenvironment. Or, the earpiece may signal the user's electronic device orsend a signal to a server to reproduce the sound of a bell. As the userhears the bell through the earpiece, a signal may be sent from theearpiece to the game to change the image on the phone to a cathedral(the image that corresponds to the sound of the bell).

FIG. 5 displays the wireless earpieces 120, also labeled earpieces 120.FIG. 5 illustrates the earpiece sensors, which may include severaldifferent forms of sensors, to be used together or individually,including but not limited to inertial sensors, accelerometers,gyroscopes, photoelectric sensors, etc. The sensors may transmitdifferent information to the earpieces, including but not limited tosoftware instructions, data transmission, wireless intrabodycommunication, biomedical monitoring, etc. The sensors may track theheart rate, steps, or other bodily movements and activity of the userthroughout the game. The sensors may be configured to work with theearpieces according to certain user preferences. For example, thesensors may be preset to work only when the user is playing the game.They may be preset to work only via galvanic communication, or only onbattery power. The earpieces may transmit data from the sensors to theuser via audio messages, or by sending messages to the user's electronicdevice. The earpieces may also transmit data from the sensors to smartglasses, where the user may read the data across the screen of theglasses. The earpieces may communicate with the sensor in differentways, depending on the configuration of the sensor. FIG. 6 displays auser 106 wearing the earpieces, and the earpieces communicating with aphone 104 and computer. Other electronic devices, though not included inFIG. 6, may communicate with the earpieces 120.

FIG. 7 shows another embodiment of an enhanced gaming experience, as thegame sound 110 is positioned within a three-dimensional audioenvironment. FIG. 7 depicts the simulation of a three-dimensional audioenvironment. The earpieces 120, worn by the user 106 (not pictured), aresurrounded by multiple sound sources. The embodiment is not limited tothe three sources displayed in FIG. 7. The three sources occupy threedifferent planes in space, displayed in the figure as the X, Y, and ZPLANES. The game sound 110 in the Y-plane appears to the user 106 tocome from the game sound source 76. The environmental sound source 74gives the illusion of the environmental sound 108 emanating from theZ-plane. And an alternate audio sound 107 appears to come from thealternate sound source 72 in the X-plane. By performing soundlocalization within the x, x-y or alternately the x-y-z planes, the useris provided with a more immersive experience. The audio signal istransduced to provide a three-dimensional audio experience for the user.The audio sounds appear to come from above, behind, and below the user.The effect of tricking the user's brain or simulating actualthree-dimensional (“3D”) audio may be created with game sounds comingfrom the phone or another electronic device (iPAD, tablet, computer,etc.), in combination with sounds coming from the user's wearableearpieces, and from the actual outside environment, enhanced with thewireless earpieces. The effect may occur via speakers inside theearpieces which use head-related transfer function, HRTF filters, crosstalk cancellation techniques, and other devices that mimic theappearance of sound waves emanating from a point in the user'sthree-dimensional space. FIG. 3 depicts one mechanism by which thewireless earpieces may be used to create this three-dimensional audioeffect. The external microphone 122 brings in the noise of the outsideworld or the user's surroundings, creating an environmental sound 108.At the same time, the internal or bone microphones 124 may convey thegame sound 110 and an alternate audio sound 107. A three-dimensionalaudio transparency effect is created as the game sound 110 is overlaidwith the environmental sounds 110 and the alternate audio sound 107. Ifenvironmental sounds 108 are combined with game sounds 110, atwo-dimensional audio effect can be created as well.

In another embodiment, shown in FIG. 8, the audio and imaging datastreams are separated before they reach the user 106. This separation ofaudio and visual streams creates, as described in the second set ofindependent claims, creates an augmented reality. The camera 98 of thephone 104 would be detached so that it could not immediately transferinformation to the phone in conjunction with the audio system. Forexample, if the camera 98 was pointed toward the ground and the user wasexpecting a certain sound to emanate, the separation of audio and visualstreams would allow the user to turn his/her/its head and pick up audiothat would not necessarily be the same as that which would be detectedby the camera-based phone position. The earpieces 120 may containinertial sensors 404 to detect the user's head movements and generate asignal to emanate a certain sound. With the user's fixed point ofreference created by disconnecting the camera, the earpieces allow theuser, with a turn of the head (user head movement 305), to detectdifferent audio streams from the point of view of the inertial sensorsin the earpiece. The earpiece settings allow the user 106 to distinguishthe camera-based outputs from the augmented audio reality whichincorporates the audio transparency and sound features. This separationof audio and visual reality may also be created via smart glassessharing data with the earpieces. The smart glasses may change the user'sview of the game, or incorporate environmental images into the screen ofthe glasses. Overlapping the visual images in the game with imagesportrayed in the smart glasses may correspond with the overlap of audioeffects between the earpieces, and the environment and game sounds. Thisoverlap of visual and audio images created by the smart glasses and thewireless earpieces may create a different but enhanced gaming experiencefor the user.

Another embodiment of the enhanced gaming experience, depicted in FIG.9, utilizes inertial sensors which measure and report the movement ofthe user. Such sensors may indicate measurements including but notlimited to the user's specific force, angular rate, magnetic field, etc.The inertial sensor(s) 404 may be located on the earpiece, or anotherwearable device of the user. As the inertial sensor 404 detects the usermovement, the sensor may relay a signal to the phone to generate a gamesound 110 based on the user movement. The inertial sensor 404 may alsorelay a signal 303 to the microphones 300 of the earpiece 120 or otherwearable device to generate a sound. One or multiple inertial sensors404 may be used to track the head position of a user (depicted in FIG.10 as user head movement 305) and to generate a game or environmentalsound based on the head position of the user. Examples of inertialsensors 404 include but are not limited to accelerometers, gyroscopes,magnetometers, etc. The inertial sensors may be used with the earpiecesand the gaming system to create a unique gaming experience for the user.As the user moves his/her head, the inertial sensors may send signals tothe wireless earpieces. The earpieces may generate sounds when theuser's head is not turned toward the game. Or the earpieces may generatesounds as the user moves and controls the image within the game.

Therefore, various methods, systems, and apparatus have been shown anddescribed. Although specific examples and embodiments are set forthherein it is to be understood that the present invention contemplatesnumerous variations, options, and alternatives.

What is claimed is:
 1. A method of providing an enhanced gamingexperience to a user, the method comprising steps of: mapping a gamesound source to a physical position; determining a physical userlocation of the user; sensing environmental audio with at least onemicrophone of an earpiece; generating a game sound, the physicalposition mapped to the game sound source, and the physical location ofthe user to provide spatial context as to where the game sound source islocated relative to the user within a three-dimensional audioenvironment; combining the game sound positioned within thethree-dimensional audio environment with the environmental audio toprovide a composite audio signal; and transducing the composite audiosignal at the earpiece to provide a three-dimensional audio experience.2. The method of claim 1 further comprising sensing movement of the userwith at least one inertial sensor of the earpiece.
 3. The method ofclaim 2 wherein the generating the game sound is based at least in parton the user movement.
 4. The method of claim 1 further comprisingtracking head position of the user using at least one inertial sensor.5. The method of claim 4 wherein the generating the game sound is basedat least in part on the head position of the user.
 6. A method ofproviding an enhanced gaming experience to a user, the method comprisingsteps of: mapping a game sound source to a physical position;determining a physical user location of the user; sensing environmentalaudio with at least one microphone of at least one of a first earpieceand a second earpiece; generating a game sound, the physical positionmapped to the game sound source, and the physical location of the userto provide spatial context as to where the game sound source is locatedrelative to the user within a three-dimensional audio environment;combining the game sound positioned within the three-dimensional audioenvironment with the environmental audio to provide a first compositeaudio signal for the first earpiece and a second composite audio signalfor the second earpiece; and transducing the first composite audiosignal at the first earpiece and the second composite audio signal atthe second earpiece to provide a three-dimensional audio experience. 7.The method of claim 6 further comprising sensing movement of the userwith at least one inertial sensor of at least one of the first earpieceand the second earpiece.
 8. The method of claim 7 wherein the generatingthe game sound is based at least in part on the user movement.
 9. Themethod of claim 6 further comprising tracking head position of the userusing at least one inertial sensor of the first earpiece and the secondearpiece.
 10. The method of claim 9 wherein the generating the gamesound is based at least in part on the head position of the user.
 11. Amethod of providing an enhanced gaming experience, comprising:maintaining an augmented reality video environment associated with acamera and video display device; maintaining an augmented reality audioenvironment associated with a set of earpieces comprising microphonesand speakers; generating a gaming experience using the augmented realityvideo environment and the augmented reality audio environment; whereinthe maintaining of the augmented reality audio environment is performedby: mapping a game sound source to a physical position, determining aphysical user location of the user, sensing environmental audio with atleast one microphone of the set of earpieces, generating a game sound,the physical position mapped to the game sound source, and the physicallocation of the user to provide spatial context as to where the gamesound source is located relative to the user within a three-dimensionalaudio environment, combining the game sound positioned within thethree-dimensional audio environment with the environmental audio toprovide a composite audio signal, transducing the composite audio signalat the earpiece to provide a three-dimensional audio experience for theaugmented reality audio environment.
 12. The method of claim 10 whereinthe set of earpieces further includes one or more inertial sensors. 13.The method of claim 12 wherein head position of a user is sensed usingthe one or more inertial sensors and wherein the head position is usedin maintaining the augmented reality audio environment.
 14. The methodof claim 11 wherein the augmented reality video environment includesgame images.
 15. The method of claim 14 wherein at least a portion ofthe game images within the augmented reality video environmentcorrespond with the game sound present in the augmented reality audioenvironment.
 16. The method of claim 14 further comprising selecting agame image based on the physical location of the user and displaying thegame image to the user based on the physical location of the user.